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PREFACE 


This  report  was  created  for  the  F-16  Aircrew  Training  De¬ 
velopment  Project  contract  no.  F02604-79-C8875  for  the  Tactical 
Air  Command  to  comply  with  the  requirements  of  CDRL  no.  B006. 

The  project  entailed  the  design  and  development  of  an  instruc¬ 
tional  system  for  the  F-16  RTU  and  instructor  pilots.  During  the 
course  of  the  project,  a  series  of  development  reports  was  issued 
describing  processes  and  products.  A  list  of  those  reports 
follows  this  page.  Tne  user  is  referred  to  Report  No.  34,  A 
Users  Guide  to  the  F-16  Training  Development  Reports,  for  an 
overview  and  explanation  of  the  series,  and  Report  No.  35,  F-16 
Final  Report,  for  an  overview  of  the  Instructional  System  De¬ 
velopment  Project. 
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EXECUTIVE  SUMMARY 


The  F-16  Instructional  Systems  Development  (ISDj)  effort 
represents  the  most  comprehensive  application  of  ISD  ^principles 
and  procedures  for  pilot  training  yet  conducted  within  the  Air 
Force.  In  order  to  avoid  the  shortcomings  and  problems  of 
previous  Air  Force  applications  of  ISD,  a  careful  analysis  of 
past  Air  Force  ISD  programs  has  been  accomplished  to  determine: 
(1)  the  lessons  which  have  been  learned  that  relate  to  the  F-16 
efforts  and  (2)  those  problems  that  have  presented  the  most 
difficulty  in  the  application  of  ISD  technology  to  pilot 
trailing.  This  report  summarizes  the  results  of  that  analysis. 

^Interviews  were  conducted  with  ^SIJ/ personnel  from  A-7,  A-10, 
F-15,  and  F-4  communities.  Conclusions  drawn  from  the  results  of 
the  interviews  are  as  follows: 


1.  Ensure  command  support  and  understanding  of  F-16  ISD 
programs  and  principles  is  maintained^. 

2.  Man  ISD  teams  adequately  and  with  trained  ISD  specialists. 

3.  Document  all  F-16  efforts  including  rationales  for  all 

decisions  and  policies  for  use  in  future  programs. 

/ 

4.  Provide  sufficient  support  for  data  handling  and  access, 
i.e.,  a  dedicated  ISD  ADP  system. 

5.  Establish  a  joint  ISD/FLIT  team  to  coordinate  and  smooth  the 
way  from  Fighter  Lead-in  Training  to  Combat  Crew  Training 
(CCT)  squadrons. 

6.  Tactics  training  wiil  be  a  problem  area.  Experimentation 
and  use  of  other  fighter  aircraft  experience  will  be 

necessary. 

/ 

7.  Review  thoroughly  the  A-10/F-15  ISD  experience  for  insight 
in  establishing  the  line  of  division  between  CCT  subjects 
and  continuation  training  subjects. 

8.  Special  attention  should  be  paid  to  alternate  training 
methods  to  substitute  for  the  unavailability  of  trainers  in 
the  initial  phases  of  the  F-16  training  project.  J 

9.  Objective  evaluation,  (CROs  and  CRTs),  must  be  developed  in 
close  cooperation  with  STAN/EVAL  personnel. 
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1.0  INTRODUCTION 


1 . 1  Purpose 

The  F-16  Instructional  Systems  Development  (ISD)  effort 
represents  the  most  comprehensive  application  of  ISD  principles 
and  procedures  for  pilot  and  instructor  training  that  has  ever 
been  conducted  within  the  Air  Force.  During  this  effort,  the  ISD 
team  hopes  to  avoid  the  shortcomings  of  previous  Air  Force 
applications  of  ISD.  An  essential  first  step  in  ensuring  this 
has  been  the  careful  analysis  of  previous  USAF  ISD  efforts  to 
determine:  (a)  the  lessons  which  have  been  learned  through 

experience  that  are  applicable  to  the  F-16  effort  and  (b)  those 
problems  that  have  presented  the  most  difficulty  in  the 
application  of  the  ISD  technology  to  aircrew  training.  This 
report  is  intended  to  summarize  the  results  of  this  study. 

The  accumulated  experience  of  the  programs  that  have  been 
reviewed  will  provide  information  highly  beneficial  to  the  F-16 
ISD  program.  Problem  areas  in  previous  projects  will  be  avoided 
through  careful  planning,  and  areas  of  strength  in  previous 
projects  will  be  studied  and  built  upon. 

In  addition,  another  positive  result  of  performing  this 
review  has  been  the  establishment  of  open  lines  of  communication 
with  currently  active  ISD  teams  within  the  USAF  aircrew  training 
community.  All  have  been  openly  and  enthusiastically  supportive 
of  the  F-16  program,  and  a  positive  attitude  of  mutual  support 
has  been  established.  This  will  not  only  aid  the  F-16  program, 
but  it  is  hoped  that  the  other  ISD  teams  within  the  Air  Force 
will  also  benefit  from  the  F-16  experience,  thus  enhancing  their 
individual  efforts. 

1 . 2  Scope 

The  ISD  programs  that  have  been  reviewed  in  preparing  this 
report  were  selected  on  the  basis  of  their  direct  applicability 
to  the  F-16.  The  review  covered  all  aspects  of  the  ISD  process, 
from  the  technical  concerns  for  instructional  design  and 
engineering  of  psychological  variables  to  the  practical  problems 
of  implementing,  managing,  and  maintaining  instructional  systems 
in  a  real-world  operational  environment.  Since  the  F-16  project 
will  include  activities  at  all  points  of  this  spectrum,  data  were 
gathered  on  as  many  relevant  issues  as  could  be  identified. 
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1.3  Methods  and  Participants 

The  task  of  conducting  this  review  was  divided  into  the 
following  three  parts: 

(1)  Development  of  the  review  questions  and  interview 
guideline . 

(2)  Identification/selection  of  ISD  teams  to  participate. 

(3)  Conduct  of  the  reviews. 


1.3.1  Development  of  Review  Questions 

The  primary  objective  in  development  of  the  questions  to  be 
used  in  the  review  effort  was  to  develop  an  interview  guideline 
that  would  provide  sufficient  information  on  past  ISD  efforts 
across  all  areas  of  project  concern.  Since  the  F-16  project 
involves  teams  of  instructional  psychologists  and  technologists, 
training  systems  specialists,  and  computer  support  specialists, 
questions  from  each  of  these  areas  were  written  and  incorporated 
into  the  final  interview  guidline. 

The  final  questions  generated  were  the  result  of 
consultations  among  the  contractor's  team  and  were  based  on  the 
following  considerations: 

(a)  A  detailed  review  of  the  ISD  process. 

(b)  The  contractor's  past  experience  in  ISD. 

(c)  Suspected  weakness/problem  areas  in  past  ISD  efforts. 

(d)  Suspected  strentghs  of  previous  ISD  efforts. 

1.3.2  Identification/Selection  of  the  Systems  to  be 
Reviewed 

Early  in  the  F-16  ISD  program  effort,  previous  Air  Force 
aircrew  training  ISD  programs  were  identified  as  candidates  for 
review.  These  included  the  A-7,  A-10,  F-4,  and  F-15.  It  was 
decided  that  the  review  effort  could  best  be  concentrated  on 
existing  Tactical  Air  Command  (TAC)  fighter  aircraft  systems, 
which  were  directly  relatable  to  the  F-16  system,  as  stated 
above . 

Having  identified  the  Systems  to  be  visited,  visit  schedules 
were  arranged  with  the  4444th  Operations  Squadron  at  Luke  AFB  for 
the  F-4  and  F-15,  and  at  Davis-Monthan  AFB  for  the  A-7  and  A-10 
systems . 
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1.3.3  Conduct  of  the  Reviews 

During  the  review  visits,  interviews  were  conducted  with 
current  ISD  team  members  by  using  the  interview  guideline 
dicussed  in  detail  in  the  following  sections  of  this  report. 

During  the  conduct  of  an  interview,  the  ISD  team 
representative  was  asked  each  question  on  the  guideline,  and  his 
response  was  recorded  both  on  tape  and  in  the  interviewer's 
notes . 

The  interviewee  was  encouraged  to  expand  and  elaborate  upon 
the  questions  posed  with  further  detail  and  clarification.  When 
the  interviewer  detected  that  information  potentially  applicable 
to  the  F-16  problem  might  be  available,  probing  questions  were 
used  and  the  respondent  was  invited  to  comment  further. 

In  addition  to  interview  notes  and  tapes,  documentation  was 
collected  from  each  ISD  team  where  available.  Documents 
solicited  included  syllabi,  media  selection  documents,  sample 
task  analyses,  sample  instructional  materials,  criterion- 
referenced  objectives  (CROs)  criterion-referenced  tests  (CRTs), 
sample  management  tools,  and  other  ISD-generated  documents.  Many 
of  these  were  collected,  although  the  total  amount  available  was 
not  as  large  as  expected. 


2.0  INTERVIEW  RESULTS 


2.1  Introduction 

The  results  of  the  interviews  on  previous  ISD  programs  are 
presented  in  the  following  pages  within  the  context  of  the 
interview  guideline.  Each  question  asked  during  the  interview  is 
stated,  followed  by  a  composite  summary  of  the  responses  across 
the  four  ISD  programs  reviewed.  A  summary  of  document  reviews  is 
not  contained  in  this  report.  The  detail  required  for  such  a 
review  would  be  unwieldy  and  would  comsume  inordinate  amounts  of 
time.  The  result  of  the  document  review  will  appear  as  elements 
in  the  design  of  F-16  project  materials  and  documents. 


2 . 2  Interview  Data  Summary 

1.  During  your  system  design,  development,  and 

implementation,  what  methods  did  you  use  for: 

A.  Task  Analysis  (TA)? 

The  TAs  for  the  A-7  and  A-10  were  developed  based  upon  the 
same  approach  specified  by  a  contractor.  A  copy  of  the  A-7 
program  report  has  been  acquired  and  is  under  review.  The  F-15 
task  analysis  was  also  performed  by  a  contractor.  There  have 
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been  many  problems  with  it  since  it  is  system-  and  equipment- 
oriented  rather  than  performance-oriented.  The  F-4  task  analysis 
is  based  on  the  A-7  approach;  however,  no  contractor  assistance 
was  included  in  the  development  of  the  F-4  program. 

B.  Definition  of  training  objectives? 

Generally,  these  have  been  developed  by  a  "committee 
consensus"  approach  using  instructor  pilot  subject  matter  experts 
(SMEs)  and  ISD  team  members.  In  some  cases,  attempts  have  been 
made  at  writing  specific  behavioral  objectives  (SBOs).  These, 
however,  have  proven  to  be  inadequate  in  that  they  are  not  stated 
in  behavioral  terms  (e.g.,  "will  understand,"  "will  have  a 
knowledge  of,"  etc.). 

C.  Development  of  CRO/CRTs? 

In  nearly  all  cases,  ISD  teams  are  just  beginning  to  get 
involved  in  the  attempt  to  generate  CROs  and  CRTs. 

D.  Course  sequencing? 

Generally  speaking,  sequences  are  developed  along  more  or 
less  conventional  lines:  for  example,  the  F-4  and  F-15  programs 
are  very  similar  because  all  F-15  instructor  pilots  (IPs)  were 
former  F-4  IPs.  The  F-15  and  F-4  blocks  of  instruction  are 
basically  the  same  for  both  academics  and  aircrew  training  tasks. 

E.  Media  trade-off /selection? 

Except  for  the  A- 7  report,  documentation  or  knowledge  of  how 
this  process  originally  took  place  is  not  known  by  the  present 
ISD  team  members. 

2.  What  data  collection  forms  (formats)  and  procedures 

were  used?  May  we  have  copies? 

The  same  data  formats  and  procedures  were  used  for  the  F-4, 
A-10,  and  F-15  as  for  the  A-7  initial  effort.  Sample  task 
training  analysis  forms  were  collected  along  with  the  "A-7 
Training  Analysis  Guide"  which  contains  additional  forms  and 
procedures . 

3.  What  was  your  approach  in  terms  of  the  task  analysis? 

Did  you,  for  instance,  use  one  analyst  and  three  SMEs? 

How  many  SMEs  were  involved? 

The  A-7  was  the  only  program  where  a  contractor  analyst  was 
involved.  The  analyst  conducted  sessions  in  which  at  least  three 
SMEs  participated.  The  F-4  and  A-10  programs  used  three  to  four 
team  members  at  any  one  time  working  as  a  committee. 

The  F-15  TA  was  originally  performed  by  the  contractor  at 
his  facility.  This  was  a  "one-shot"  effort  with  no  requirement 
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for  update.  The  analysis  was  performed  before  the  aircraft  was 
flying  and  was  based  largely  on  everyone's  “best  estimate"  at  the 
time  of  what  the  aircraft  coaid  do  and  how  it  would  be  employed. 
Therefore,  the  task  analyse  was  tactically  outdated  by  the  time 
the  aircraft  arrived. 

There  were  no  more  than  three  to  four  F-15  ISD  team  members 
at  any  one  time.  The  team  members  had  to  divide  their  time 
between  the  ISD  effort  and  flying. 

4.  What  data  management  forms  did  you  use  for: 

A.  TA  storage? 

The  A- 7  program  procedures  were  followed  in  all  cases.  The 
task  analysis  is  input  to  the  computer  based  upon  the  data  from 
the  "Task  Training  Analysis"  sheet.  This  has  proven  to  be  more 
detailed  than  necessary. 

B.  TA  retrieval? 

Across  the  ISD  programs,  access  to  the  data  base  was  a 
problem.  Those  who  know  how  to  program  the  computer  system  for 
retrieval  are  gone.  Problems  with  the  fundamental  computer 
program  also  exist.  Computerized  TA  storage  and  retrieval  are  so 
difficult  as  to  be  virtually  beyond  the  reach  of  the  average  ISD 
team  with  no  computer  expertise. 

C.  TA  sort? 

Again,  the  A- 7  program  sort  capabilities  and  procedures  are 
used.  The  sort  capability  is  rather  extensive  in  that  one  can 
"sort  on  any  heading"  (from  the  "Task  Training  Analysis"  sheet) 
that  is  inserted  in  the  computer  (e.g.,  tasks  per  media,  media 
capability  to  train  a  task,  etc.).  As  in  TA  retrieval,  the 
procedures  are  not  used  because  they  would  require  much  time  and 
an  extensive  knowledge  of  computer  programmng  and  specific 
computer  programs . 

5.  How  did  you  handle  program  update  and  revision  of: 

A.  System  constraints? 

B.  Aircraft  (A/C)  capabilities  and  missions? 

C.  Data  support  requirements/programs? 

All  projects  reviewed  are  attempting  to  maintain  their 
present  instructional  programs  and  have  little  time  for 
innovation  and  change.  Only  A/C  system  changes  are  reflected  in 
the  update  of  training  materials.  Such  system  changes  are,  of 
course,  more  prevalent  in  the  A-10  and  F-15  than  in  the  older  A-7 
and  F-4  programs. 
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Primarily  revisions  are  being  driven  arbitrarily  by  higher 
headquarters  direction  and  requirements  without  consideration  of 
ISD  planning  and  methodology.  One  comment  was,  “What  is  required 
is  stronger  headquarters  support  of  the  ISD  approach  and  a  better 
understanding  of  ISD  requirements."  This  remark  was  intended  to 
mean  that  additional  manpower  on  ISD  teams  was  needed  and  that 
mandated  changes  needed  to  reflect  greater  insight  into  the 
processes  and  effects  of  ISD. 

6.  What  approach/procedures/formats  did  you  use/are  you 

using  for: 

A.  Program  update? 

B.  Student  testing  and  evaluation  (both  academics, 
nonflight  and  flight,  including  procedures  and 
formats  for  student  grading/performance)? 

As  far  as  updating  is  concerned,  the  programs  which  have 
been  reviewed  are  being  updated  as  directed  by  TAC  Headquarters. 
All  programs  reviewed  are  currently  using  the  TAC  standard  0  to  4 
grading  system.  Task  standard  is  a  grade  level  of  2  (i.e.,  "He 
recognizes  and  corrects  errors"),  with  a  grade  level  of  3 
required  for  critical  tasks. 

For  the  F-15,  students  go  through  emergency  procedures 
training  every  two  weeks  in  the  cockpit  procedures  trainer  (CPT), 
where  their  performance  is  evaluated. 

7.  What  do  you  see  as  the  "key"  problems  in  applying  ISD 

to  new  flying  training  programs? 

The  responses  to  this  question  can  be  summarized  as  follows: 

A.  For  emerging  systems,  such  as  the  A-10  and  F-15, 
the  data  available  are  not  firm.  Procedures  are 
not  firm.  Performance  data  on  the  A/C  are  not 
reliable,  and  there  is  an  insufficient  data  base. 
In  the  case  of  the  F-15,  this  resulted  in  a 
complete  set  of  sound/slide  programs  that  were  out 
of  date  on  delivery  due  to  changes  in  the  "Dash 
One"  manual  procedures. 

B.  Extensive  system  changes  and  changes  in  related 
procedures,  especially  during  the  first  year, 
caused  the  F-15  program  many  problems. 

C.  There  is  a  problem  in  keeping  pace  in  the  syllabus 
between  the  ground  training  and  flying  training 
due  to  the  small  number  of  A/C  and  A/C 
availability  at  the  beginning. 

D.  Presently  F-15  ground  train,  iq  in  being  driven  by 
the  flying  training  schedule,  ux ?ing  students 
insufficient  ground  training  i  '  before  they  fly. 
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E.  The  lack  of  a  simulator  early  in  the  program  was  a 
problem  for  both  the  A-10  and  the  F-15.  Without 
the  simulator,  the  "systems"  training  in  terms  of 
dynamic  feedback  was  a  problem.  F-15  finally 
acquired  videotapes  depicting  the  system 
performance,  which  were  very  successful. 

F.  Many  problems  were  experienced  with  the  impact  of 
personalities  on  the  program,  i.e.,  commanders  and 
their  particular  wishes.  Some  team  members  felt 
the  need  for  a  better  documentation  of  the  ISD 
processes  and  rationale  to  "fend  off  irrational 
changes  to  the  program" . 

8.  If  you  could  start  anew  on  your  particular  ISD 

program,  what  would  you  recommend  to  improve  the 

application  of  the  ISD  approach? 

Responses  to  this  question  were  varied.  However,  the 
following  is  an  applicable  summary  across  systems: 

A.  It  is  critical  that  complete  and  detailed 
documentation  be  developed  and  maintained  for 
decisions  made  during  design  and  development. 

B.  ISD  teams  should  be  more  adequately  staffed, 
especially  with  ISD-trained,  subject-matter 
qualified  personnel. 

C.  There  should  be  a  command-wide  education  program 
on  ISD  in  order  to  provide  a  more  positive 
understanding  and  support  of  ISD  efforts. 

D.  In  the  case  of  emerging  systems  (A-10/F-15),  those 
interviewed  felt  that  they  should  not  have  gone  to 
audio-visual  (AV)  materials  as  early  as  they  did 
because  of  system  changes.  The  A-10  program  did 
try  to  prevent  this  problem  by  avoiding  inclusion 
of  procedural  tasks  in  sound/slide  programs. 

There  was  a  general  feeling  that  easily  revised 
written  materials  (e.g.,  workbooks)  early  in  the 
program  would  be  a  better  medium  for  instruction 
in  an  emerging  A/C  system. 

E.  Both  A-10  and  F-15  interviewees  felt  that  tactics 
will  be  a  block  of  instruction  that  will  be  very 
difficult  to  handle  because  of  "experimenting 
during  the  initial  stages  of  the  program."  The 
F-16  joint  test  force  (JTF)  will  be  able  to 
provide  some  input,  but  members  of  the  JTF  have 
usually  been  out  of  the  fighter  community  for  some 
time  and  are  not  necessarily  knowledgeable  about 
current  tactical  scenarios. 
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The  F-15  community  engaged  in  extensive  experimenting  with 
tactics  after  the  aircraft  began  to  arrive  at  Luke.  They  feel 
that  F-15  and  F-16  tactics  should  be  fairly  similar  and  their 
efforts  should  provide  a  valuable  data  base  for  F-16. 

9.  What  was  your  data  management  approach  (automated  data 
handling)?  Were  you  keyed  to  the  existing  TAC  B-3500 
system? 

All  systems  reviewed  are  keyed  to  the  existing  TAC  B-3500 
computer  system.  The  biggest  problem  appears  to  be  access  to  the 
data  base,  since  another  agency  on  base  must  be  dealt  with  in 
order  to  enter/ retrieve  data.  The  opinion  was  that  what  is 
needed  is  a  more  responsive,  dedicated  system  with  at  least  one 
assigned  programmer  to  maintain,  update,  and  access  the  data  base 
in  response  to  ISD  team  needs. 

10.  Did  you  find  in  your  recommended  (preferred)  approach 
that  you  ran  into  problems  with  existing  AF/TAC 
training  doctrine/procedures? 

The  answer  to  this  question  was  “yes"  across  all  systems 
reviewed.  It  is  apparent  that  existing  regulations  do  not 
recognize  the  requirements  of  ISD;  they  reflect  a  lack  of 
understanding  of  ISD  and  the  systematic  approach. 

The  A-10  team  had  problems  regarding  a  rewrite  of  the  55  and 
51  series  documents  to  bring  them  more  realistically  in  line  with 
the  A-10  system  and  its  tactical  employment. 

It  was  also  mentioned  that  a  greater  effort  to  align 
training  (based  on  the  ISD  approach)  and  standard  evaluation 
(STAN/EVAL)  requirements  is  required  in  order  to  attempt  to 
alleviate  disparities  between  performance  evaluation  techniques. 

11.  We  would  like  to  have  any  copies  of  actual  program 
documentation,  such  as: 


A.  Syllabus. 

B.  Media  selection. 

C.  Sample  formats  used  in  implementing  your  program, 
i.e.,  TAs,  CROs,  CRTs,  scheduling  determination, 
etc . 

D.  Instruction  materials. 

E.  Goal  Analysis. 


Syllabi  were  obtained  from  all  four  communities.  Where 
available,  phase  manuals  and  instructor  guides  were  also 
obtained . 
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Very  little  documentation  exists  for  media  selection,  with 
the  exception  of  the  A-7  program  document. 

A  sample  of  the  A-7  program's  first  cut  at  CROs  was  obtained 
and  is  being  reviewed . 

12.  Please  feel  free  to  give  us  any  general  comments 
regarding  ISD,  its  application  and  problems. 

A  key  response  across  the  systems  reviewed  was  a  "lack  of 
command-wide  support  and  understanding."  In  addition,  there  was 
consensus  that  there  is  a  lack  of:  (a)  personnel  trained  in  the 
ISD  process,  (b)  funding  and  proper  facilities  for  media 
development  and  support,  and  (c)  time  for  all  schedule 
commitments . 

One  response  was,  "How  comprehensive  do  you  want  to  be? 

What  do  you  want  out  of  the  program?  If  ISD  is  to  be  done 
properly,  it  requires  a  realistic  allocation  of  the  three  items 
stated . " 

13.  How  early  in  the  planning  process  did  your  ISD  people 
get  involved? 

A.  Prime  system  planning  and  design? 

B.  Maintenance  planning  and  approach  design  (A/C 
availability ) ? 

C.  System  support  planning  and  approach  design? 

Probably  of  most  importance  here  are  the  responses  of  the 
A-10  and  F-15  communities,  since  their  experiences  are  more 
aligned  with  what  we  will  experience  with  the  F-16  system. 

In  the  case  of  the  A-10,  the  ISD  team  was  formed  well  before 
A/C  delivery,  using  the  A-7  IPs.  For  the  F-15,  three  to  four  F-4 
IPs  were  chosen  early  in  the  program  and  appointed  as  the  F-15 
ISD  team  prior  to  A/C  arrival. 

In  both  cases,  the  teams  had  to  deal  with  "second-hand" 
knowledge.  They  were  not  able  to  gain  as  much  interface  with  the 
JTF  as  desired,  and  they  did  not  have  an  opportunity  to  fly  the 
aircraft.  For  the  F-15,  some  visits  to  the  factory  were  made. 

14.  What  problems  arose  from  getting  the  system  design  and 
development  phases  late?  How  did  you  handle  these 
problems? 

Here  again,  responses  from  the  A-10  and  F-15  interviewees 
are  most  appropriate.  Their  biggest  problem  was  the  development 
of  new  tactics.  In  doing  so,  they  (a)  took  inputs  from  JTF 
personnel  and  (b)  studied  available  documents  on  employment  of 
the  A/C.  From  these  they  made  an  initiai^guess  at  appropriate 
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tactics,  flew  F-4  type  tasks,  and  determined  how  they  could 
improve  them  based  on  performance  of  the  A-10/F-15  and  its 
systems.  Over  the  course  of  the  first  two  or  three  classes,  they 
made  tactics  section  revisions. 

15.  To  some  degree,  tactics  development  always  waits  for 
the  A/C  delivery  and  initial  tactical  training 
sufficient  to  get  a  background  of  experience  with  the 
particular  A/C.  How  do  you  handle  the  development  of 
the  tactics  part  of  the  training  to  get  maximum 
development  efficiency? 

Responses  to  this  question  are  covered  under  question  14  and 
other  previous  questions  and  responses. 

16.  How  did  you  handle  problems  of  deciding  how  much 
training  to  eliminate  by  selection  (e.g.,  only  pilots 
with  operational  experience  in  similar  A/C  or 
whatever),  and  how  much  of  the  required  training  would 
be  conducted  in  the  combat  crew  training  squadron 
(CCTS)  and  how  much  put  off  to  be  carried  over  later 
in  the  operational  squadron? 

Early  in  the  F-15  program,  it  became  apparent  that  due  to 
aircraft  and  sortie  availability,  the  "mission-ready"  graduate 
concept  could  not  be  attained.  Consequently,  TAC  headquarters 
guidance  was  requested  as  to  what  graduates  should  be  capable  of 
doing.  The  decision  was  that  the  F-15  program  should  be  geared 
to  produce  a  "safe"  student  who  could  practice  to  mission-ready 
criteria  within  the  operational  unit.  This  guidance  applied  to 
experienced  fighter  pilots  only.  At  the  time  of  this  interview, 
the  undergraduate  pilot  training  (UPT)  graduate  program  for  the 
F-15  had  not  been  completely  decided  and  will  depend  upon  the 
particular  fighter  lead-in  training  received  and  the  availability 
of  F-15  aircraft. 

In  the  case  of  the  A-7  and  F-4,  the  attempt  was  and  is  to 
bring  the  student  as  close  to  mission-ready  as  possible.  The 
student  actually  qualifies  as  mission-ready  in  the  operational 
squadron  follow-on  (continuation)  training. 

The  A-10  personnel  feel  they  had  the  same  problem  we  will 
have  in  the  F-16.  When  it  came  to  tactics,  they  "ran  into  a 
stone  wall"  because  there  was  not  an  operational  squadron  in 
existence,  and  the  line  of  division  between  CCTS  training  and 
operational  squadron  (continuation)  training  could  not  be  clearly 
defined.  However,  it  was  felt  that,  ideally,  formal  training 
(CCTS)  should  mirror  continuation  (operational  squadron) 
mission-ready  requirements  as  much  as  possible. 

17.  Did  you  use  system  operation  task  analysis  data, 
mission  performance  task  analysis  data,  or  both? 
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Responses  across  the  four  communities  varied.  For  the  F-15 
and  F-4,  both  kinds  of  analysis  were  performed.  For  the  A-7,  a 
mission-oriented  task  analysis  was  carried  out.  In  the  case  of 
the  A-10,  a  mission  analysis  was  intended,  but  resource 
constraints  resulted  in  the  use  of  the  contractor's  task  listing 
only,  which  was  basically  system  operation  oriented. 

18.  How  did  you  check  reliability  and  validity  of  student 
performance  evaluation  techniques? 

In  general,  formal  assessment  and  use  of  student  evaluation 
information  are  weak.  In  the  A-7  community,  fail  items  and  grade 
slips  are  used  to  try  to  revise  academics  and  syllabi.  The  A-10 
program  is  attempting  to  develop  a  more  objective  evaluation 
program.  No  objective,  procedure-oriented  program  currently 
exists  in  the  F-4  or  F-15  programs. 

19.  Did  the  performance  of  the  initial  students 
(especially  if  they  were  the  "cream  of  the  crop") 
provide  an  adequate  basis  for  evaluating  the 
instruction  for  later  classes  (UPT  student,  for 
example ) ? 

In  the  F-4,  the  program  was  ongoing,  and  the  initial 
students  were  no  different  from  those  following.  In  the  F-15,  no 
UPT  "B"  students  have  been  trained  and  deployed  yet,  so  there  is 
no  data . 

In  the  A-7  program,  changes  were  needed  in  the  training 
provided  to  later  classes,  as  the  "cream-of-the-crop"  students 
ran  out.  In  the  A-10  program,  expansions  were  made  in  parts  of 
the  curriculum  for  the  same  reason. 

20.  Were  you  able  to  close  the  feedback  loop  between 
operational  squadrons  and  the  CCTS  in  order  to  provide 
a  flow  fo  information  for  ISD  correction/improvement? 

All  programs  are  attempting  to  close  the  loop.  All  have 
problems  of  one  kind  or  another.  In  the  F-4  community,  only  a  30 
percent  return  rate  has  been  attained  for  evaluation 
questionnaires  sent  out  to  operational  commands.  The  A-7  group 
is  in  the  process  of  attempting  to  improve  this  kind  of  feedback. 
The  A-10  program  feels  the  need  for  a  more  formalized  approach 
than  the  one  it  has  now.  The  F-15  cadre  used  field  interviews 
for  follow-up  on  their  first  graduates  and  feel  they  obtained 
some  good  subjective  data.  However,  they  also  feel  the  need  for 
CROs  in  their  program  in  order  to  enable  more  objective 
evaluation  feedback  from  the  field. 

21.  How  did  you  handle  the  "halo"  effect  in  grading 
officers?  Were  you  able  to  establish  an  "oDjective" 
grading/evaluation  system? 


This  appears  to  be  a  universal  evaluation  problem.  In  every 
case,  the  indication  is  that  difficulty  of  objective  grading  is 
proportional  to  rank. 

22.  Was  the  time-line  schedule  met? 

In  every  case,  the  individuals  interviewed  believed  or  were 
sure  that  the  schedules  were  slipped  to  some  degree.  In  the  F-15 
community,  the  team  felt  it  had  met  its  own  schedules  but  was 
detained  by  outside  factors,  such  as  a  delay  in  simulator 
delivery. 

23.  Was  the  funding  exceeded? 

The  individuals  interviewed  did  not  have  data  available  to 
document  answers  to  this  question,  but  they  generally  believed 
that  funding  was  exceeded  to  some  degree . 

24.  Standardized  ISD  and  automated  data  processing  (ADP) 
terminology? 

There  was  unanimous  agreement  that  this  is  a  problem,  that 
it  is  being  addressed  by  headquarters,  and  that  it  will  probably 
be  resolved  by  a  rewrite  of  TAC  B-3500  procedures  for  ISD  and  ADP 
systems . 

25 .  What  about  students  returning  to  a  system  after  being 
away?  Do  they  go  back  through  a  "special  course"? 

In  general,  these  students  are  sent  back  through  a 
transition  course.  This  is  currently  true  for  the  F-4,  the  F-15, 
and  the  A-7 ,  depending  on  the  length  of  stay-away.  The  A-10 
system  is  new  and  the  procedure  is  not  yet  firm,  but  it  is 
assumed  that  a  tailored  transition  course  will  be  provided,  again 
depending  on  the  length  of  stay-away.  The  transition  courses 
tend  to  be  tailored  to  the  individual  needs  of  the  students  to 
aid  in  their  proficiency  advancement. 

26.  To  what  extent  was  your  training  system  self-paced? 

All  four  of  the  programs  examined  are  lockstep.  In  the  F-4 
program,  proficiency  advancement  is  possible,  but  advanced 
students  still  have  to  wait  and  graduate  with  their  class.  CPTs 
and  sound/slide  materials  are  available  in  the  A-10  program  for 
students  to  use  on  their  own  time,  but  the  program  is  lockstep. 
The  F-15  program  personnel  interviewed  believe  that  program 
efficiency  could  be  increased  by  appropriate  self-pacing 
(especially  in  the  transition  course). 

27.  Can  students  work  ahead? 

In  all  cases,  students  can  work  ahead  to  some  degree. 
Proficiency  advancement  is  possible  in  the  F-4  program;  the 
learning  center  is  made  available  to  A-7  students;  and  AV 


programs  and  the  -1  and  -34  manuals  are  available  for  the  F-15 
students.  The  F-15  personnel  interviewed,  however,  did  not  feel 
that  the  structure  of  the  academics  and  syllabus  made  such  work 
effective  because  students  achieve  no  savings  in  time.  Also,  as 
noted  above,  all  programs  are  lockstep  in  spite  of  the 
capabilities  for  students  to  work  ahead. 

28.  How  were  stragglers  handled? 

In  all  cases,  stragglers  are  allowed  three  extra  rides  to 
qualify  for  each  phase  before  being  sent  to  the  review  board.  In 
the  F-15  community,  the  only  stragglers  now  are  the  ones 
occurring  as  the  result  of  scheduling  problems,  i.e.,  the  student 
cannot  be  adequately  scheduled  to  the  necessary  academic  and  A/C 
sessions  in  the  required  timelines. 

29.  How  much  pretesting  was  used?  Was  it  effective? 

Generally,  pretesting  is  not  used  in  the  communities.  The 
F-4  community  uses  some  for  the  "backseat"  student.  However,  it 
was  agreed  across  systems  that  pretesting  would  be  desirable. 

Pretesting  would  not  be  beneficial  in  aircraft  systems 
areas.  The  student  still  needs  to  learn  the  new  aircraft  and  its 
systems.  In  air-to-air,  pretesting  might  largely  eliminate 
formal  training  in  basic  fighter  maneuvers  ( BFM)  or  in  intercept, 
depending  upon  the  student's  fighter  lead-in  training. 

30.  What  were  students'  preferences  on  lesson  types? 

Media?  Strategy? 

Responses  in  this  area  were  highly  variable .  The  A-7 
community  had  no  data  at  all  on  this  subject.  Sound/slide 
programs  were  received  in  the  A-10  program,  but  got  variable 
responses  in  the  F-15  program  where  there  was  a  wide  range  of 
quality.  "Retreads"  in  the  F-4  program  particularly  liked  the 
self-pacing  capability  of  available  sound/slide  programs  as 
compared  to  lecture  alternatives.  Students  in  the  F-15  program 
were  very  positive  about  the  videotapes  available  in  the  inertial 
navigation  system  (INS),  radar,  and  flight  control  systems  areas 
and  with  the  simulator. 

31.  Did  you  have  multiple  syllabi?  Did  they  work? 

All  programs  had  multiple  syllabi.  All  had  resulting 
scheduling  problems.  As  might  be  guessed,  the  major  scheduling 
problems  involved  limited  availability  assets,  principally  A/C 
and  simulators. 

32.  How  was  OFT  (operation  flight  trainer)  and  A/C  work 
integrated?  Was  it  done  successfully? 

In  all  cases,  the  strategy  used  was  to  integrate  simulators 
and  air  work  with  academics,  generally  proceeding  from  academics 
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to  trainer  to  aircraft  based  on  prerequisite  structures 
determined  by  some  form  of  training  analysis.  Any  lack  of 
success  in  this  approach  appeared  to  stem  from  difficulty  in 
scheduling  simulators  so  as  to  achieve  the  planned  sequences. 

That  is,  integration  of  simulator  and  aircraft  exercises  was 
successful  whenever  scheduling  allowed  the  planned  sequence  to  be 
implemented. 

33.  What  administrative  problems  were  encountered  in  using 
the  system? 

No  major  problems  were  indicated  in  any  of  the  programs  in 
the  area  of  facilities  or  in  the  updating  of  instructional 
materials.  No  instructor  training  problems  were  raised  by  F-4 
and  A-10  personnel.  In  the  A-7  program,  a  problem  was  expressed 
in  the  area  of  instructor-candidate  qualifications,  where  a  need 
for  better  prerequisites  was  felt.  In  terms  of  resources,  the 
F-4  and  F-15  programs  are  constrained  by  the  number  of  sorties 
available.  The  A-10  program  suffered  from  a  lack  of  resources  in 
all  areas. 


34.  How  were  instructors  scheduled?  What  duties  did  they 
have? 

Typically,  academic  instructors  also  fly  with  the  students. 
IPs  of  the  squadron  do  not  teach  academics. 

If  one  adhered  strictly  to  the  syllabus,  academic 
instruction  and  flying  instruction  (and  instructors)  should 
probably  be  assigned  to  the  same  squadron. 

35.  What  incentive  systems  are  built  into  the  system  you 
used? 

All  programs  tend  to  assume  that  intrinsic  incentive  is 
built  into  the  natural  competitiveness  of  fighter  pilots.  In  the 
A-7  and  F-4  programs,  rewards  help  make  this  explicit  in  the  form 
of  "top  gun"  and  "top  student"  designations  in  the  A-7  community, 
and  "top  gun,"  "top  front  seater/top  back  seater/top  overall," 
and  "top  academic  (front/back)"  designations  in  the  F-4 
community. 


36.  How  long  is  your  syllabus? 

Copies  of  syllabi  were  obtained  from  all  programs  reviewed 
and  are  a  part  of  the  F-16  ISD  data  file  for  further  review. 

37.  Do  all  students  reach  mastery?  What  happens  to  those 
who  don't? 

For  the  A-7,  F-4,  and  A-10  communities,  the  answer  is, 
generally,  yes.  However,  this  is  not  objectively  defined.  If 
there  is  a  problem  (e.g.,  because  of  a  scheduling  difficulty,  a 
sortie  is  missing),  this  is  noted  and  goes  forward  with  the 
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student's  records,  stating  the  situation  and  making  a 
recommendation  for  correction. 

The  "B"  course  (UPT)  student  is  required  to  achieve  some 
subjective  performance  level.  However,  again  the  "B"  course  does 
not  have  objective  grading  of  performance  in  all  flying  training 
tasks.  At  this  point,  the  only  objective  evaluation  is  60-2  on 
instrument  requirements.  He  has  to  be  "safe."  If  he  does  not 
qualify,  extra  rides  are  allowed  until  he  does. 

38.  Does  the  training  system  or  the  personnel  system 
define  the  student  input? 

All  communities  responded  that  the  training  squadron  has  no 
influence  on  student  input. 

39.  How  do  you  receive  feedback  from  the  operational 
squadron? 

The  response  to  this  question  has  been  covered  earlier.  The 
A-7  community  has  tried  student  evaluation  questionnaires  with 
less  than  adequate  response.  Typically,  the  questionnaire 
approach  has  not  met  with  favorable  response.  F-15  found  that 
personal  follow-up  and  interviews  proved  more  effective. 

40.  Does  the  training  squadron  develop  student 
requirements — or  is  the  number  of  students  input  to 
the  system  dictated? 

The  response  here  was  unanimous  across  systems  that  student 
input  is  determined  by  higher  authority. 

41.  Are  the  students'  grades  stored  automatically? 

No.  At  present  only  manual  grade  books  are  used  in  all  the 
systems  reviewed. 

42.  Are  grades  tied  to  objectives? 

The  response  was  the  same  across  systems.  Grades  are  tied 
to  objectives  only  in  terms  of  USAF/TAC  60-2  STAN/EVAL 
requirements.  However,  all  communities  reviewed  are  striving  to 
develop  objective  measures. 
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3.0  CONCLUSIONS  AND  RECOMMENDATIONS 


3.1  Introduction 

The  review  of  previous  ISD  Programs  and  the  resultant  data 
will  undoubtedly  prove  to  be  of  great  value  to  the  F-16  ISD  Team  in 
design,  development,  and  implementation  of  the  F-16  Training 
System. 


The  ISD  Team  members  of  all  the  programs  reviewed  were 
extremely  cooperative.  The  data  obtained  was  based  upon  personal 
experiences  in  the  application  of  the  ISD  approach.  Knowledge  of 
these  "real  world"  problems  as  they  have  occurred  in  the  past 
should  assist  the  F-16  ISD  Team  in  avoiding  or  tempering  some  of 
the  major  problem  areas  encountered  in  the  past. 

Readers  of  this  study  should  bear  in  mind  that  it  attempts 
to  gather  and  present  as  much  information  from  as  many 
experienced  persons  as  possible  within  time  and  budget 
constraints.  The  data  gathered  does  not  represent  a 
sophisticated  and  exhaustive  study  of  the  total  TAC  ISD 
experience,  employing  authorized  TAC  spokesmen  and  all  possible 
recognized  sampling  and  data  gathering  techniques  which  would  be 
employed  in  a  formal  study.  Such  was  beyond  the  scope  and  spirit 
of  what  was  intended  within  the  F-16  development  project.  Though 
much  confidence  is  placed  in  the  data  reported  herein,  it  is 
recognized  that  these  data  were  gathered  for  the  specific  purpose 
of  surveying  an  active  operational  area  prior  to  entering  into 
the  same  area  to  perform  development.  This  study  was  not 
conducted  for  the  purpose  of  recommending  major  command-wide 
changes  in  policy  or  practice.  It  is  appropriate  that  this 
report  be  suggestive  of  areas  which  deserve  further,  more  formal 
study  but  those  studies  are  yet  required,  followed  by  careful 
planning  before  change  action  is  recommended. 

The  areas  of  concern,  conclusions,  and  recommendations 
presented  in  the  following  sections  of  this  report  are  those  of 
the  contractor's  ISD  Team  members  and  should  not  be  interpreted 
as  necessarily  reflecting  an  official  view  of  TAC  personnel  or 
other  Air  Force  agencies  contacted  and  interviewed  during  this 
effort . 


3.2  Areas  of  Concern  and  Recommendations 

The  intent  of  this  section  is  to  name  areas  of  concern  which 
appear  to  be  most  relevant  to  the  F-16  ISD  effort.  The  areas  of 
concern  addressed  include  the  following: 
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Command  Support 
Manning 

Initiating  and  Maintaining  ISD  Documentation 

Data  Processing  and  Handling 

Fighter  Lead-in  Training  Output 

Tactics  Training 

CCTS  and  Continuation  Training 

Training  Alternatives  Awaiting  Simulator  Delivery 
STAN/EVAL  and  Objective  Student  Evaluation 


3.2.1  Command  Support  of  ISD  Programs 

The  review  results  show  that  there  exists  a  very  definite  need 
for  better  command-level  understanding  of  ISD  programs  and  the 
principles  built  into  ISD  products.  Instructional  systems,  like 
any  complex  system,  respond  to  internal  changes  with  a 
corresponding  change  in  output. 

The  review  of  past  ISD  programs  showed  that  revisions  to  TAC 
training  programs  are  often  driven  by  higher  headquarters 
direction.  Indications  show  that  they  are  often  not  made  through 
an  understanding  of  the  ISD  process  and  its  products  but  because  of 
demands  placed  upon  headquarters  which  result  in  sometimes 
arbitrary  changes  being  mandated  to  ISD  teams. 

Based  on  the  results  of  the  review,  a  concerted  effort  is 
recommended  on  the  part  of  the  F-16  ISD  Team  in  maintaining  the 
already  strong  education  and  information  program  at  the  TAC 
Headquarters  level  which  they  have  initiated.  This  will  increase 
the  likelihood  that  future  changes  to  the  instructional  system  will 
be  made  with  an  understanding  of  the  likely  effect  upon  the 
system's  student  output. 

3.2.2  Manning 

All  previous  ISD  programs  reviewed  stated  the  need  for 
additional  manpower  trained  in  ISD  techniques  and  fully  qualified 
experts  in  the  subject  matter.  All  programs  reviewed  stated,  that 
due  to  inadequate  levels  of  manning,  training,  and  expertise,  they 
had  little,  if  any,  time  for  innovation  and  improvement  of  the 
instructional  system.  The  major  effort  of  present  ISD  Teams  under 
present  staffing  levels  appears  to  be  maintaining  the  existing 
instructional  systems  and  keeping  them  current  in  their  present 
configurations . 
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The  following  are  recommended:  (1)  ISD  Teams  should  be  more 
adequately  manned  or  supported  to  handle  task  loads  imposed  upon 
them.  (2)  ISD  Teams  should  be  manned  with  trained  ISD  specialists, 
or  be  trained  following  assignment.  Since  there  is  a  long  lead 
time  required  for  assimilation  of  the  ISD  method  and  techniques, 
the  former  is  recommended.  (3)  ISD  Teams  should  be  manned  with 
subject-matter  qualified  personnel.  Development  experience  in  the 
Air  Force  and  other  services  as  well  demonstrate  distinct 
complications  from  use  of  non-experts. 


3.2.3  Documentation  of  ISD  Efforts 

It  was  not  possible  to  obtain  much  of  the  important 
documentation  for  the  ISD  programs  reviewed  because  in  some  cases 
it  did  not  exist  and  in  others  it  was  not  easily  obtainable.  ISD 
makes  use  of  such  documentation  extensively,  particularly  during 
the  review  of  ISD  decisions  and  policies.  In  fact,  these 
documents  form  the  foundation  upon  which  a  well-designed 
instructional  system  should  be  based.  The  lack  of  documentation 
should  be  remedied  by  insuring  that  ISD  projects  are  well 
documented  from  the  outset,  both  with  working  documents  (Task 
Analyses,  Objectives  Hierarchies,  CROs  and  CRTs,  Media  Slections, 
etc.)  and  in  terms  of  rationale  and  process  documents  which 
explain  how  decisions  were  made,  how  processes  were  applied,  and 
how  and  why  policies  were  formed. 

As  well  as  insuring  that  documentation  of  the  ISD  effort  is 
produced,  a  capability  should  be  established  to  maintain  and 
update  it.  Some  difficulty  has  been  experienced  by  the  ISD  Teams 
interviewed  in  obtaining  current  versions  of  the  working 
documents  which  do  exist  from  information  storage,  which  has 
resulted  in  a  reduced  reliance  upon  them.  An  adequate 
documentation  maintenance  system  must  proceduralize  the  detection 
and  registration  of  changes  to  documents  and  the  production  of 
current-version  documents.  It  must  also  insure  that  this  can  be 
effected  in  a  timely  manner  and  without  great  difficulty  by  the 
ISD  Team.  Without  provision  for  adequate  and  accessible 
documentation,  ISD  Teams  cannot  be  expected  to  maintain  updated 
systems  exemplifying  the  benefits  attainable  through  application 
of  ISD. 


3.2.4  Data  Processing  and  Handling 

A  lack  of  adequate  support  in  data  handling  and  access  has 
resulted  in  a  variety  of  problems  noted  in  the  previous  ISD 
programs  studied.  As  noted  in  the  previous  section,  on 
documentation  maintenance,  the  problems  in  some  cases  have  been 
extensive  enough  to  make  the  data  base  unusable  or  to  motivate 
personnel  to  leave  it  alone.  Data  bases  which  are  used  in  ISD 
permit  the  study  of  system  operation,  detection  of  needed 
changes,  and  the  update  of  the  system  in  an  organized  and 
consistent  fashion.  Maintenance  of  data  bases  on  student 
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performance  and  progress,  on  system  operation  and  costs  and 
capabilities  to  manipulate  that  data  base  to  obtain  reports  and 
projections  is  important  to  the  running  of  an  effective  and 
efficient  instructional  system. 

The  results  of  this  review  indicate  that  data  processing 
support  of  this  kind  is  not  practically  available  due  to  the 
involved  procedures  and  technical  sophistication  required  to  use 
presently  available  support.  For  an  improvement  of  presently 
operating  systems  and  for  the  implementation  of  word 
sophisticated  systems  in  the  future,  there  exists  a  need  for  a 
more  responsive,  (ideally  a  dedicated)  system  of  ADP  support  to 
meet  ISD  Team  needs. 

The  F-16  ISD  contracted  tasks  include  a  task  for  the  review 
and  evaluation  of  existing  ISD  data  automation  and  the 
development  of  recommendations  for  data  processing  and  handling 
systems  which  will  meet  the  needs. 


3.2.5  Fighter  Lead-in  Training  Output 

Data  gathered  in  the  review  confirmed  that  the  student 
output  of  Fighter  Lead-in  Training  will  have  high  impact  on  the 
content  of  the  F-16  "B"  course. 

If  Fighter  Lead-in  Training  for  students  destined  for 
assignment  to  F-16  Combat  Crew  Training  can  be  influenced  by 
identified  F-16  training  objectives,  the  transition  from  Fighter 
Lead-in  Training  to  CCTS  training  can  be  made  more  smooth  and 
less  marked  by  redundancies  and  gaps  in  training. 

This  will  require  cooperative  development  of  the  EFLIT  and 
F-16  syllabi.  That  effort  should  be  a  joint  F-16  ISD/EFLIT  team 
approach  initiated  and  coordinated  through  TAC  Headquarters. 


3.2.6  Tactics  Training 

The  results  of  the  review  show  clearly,  based  upon  the 
experiences  of  the  A-10  and  F-15  community,  that  developing 
tactics  and  tactics  training  will  be  a  problem  area  for  the  F-16 
ISD  Team. 

Initially,  a  great  deal  of  experimentation  will  be  required 
in  the  development  of  tactics.  Based  upon  the  review  data,  F-15 
and  F-16  air-to-air  tactics  should  be  similar,  and  every 
advantage  should  be  taken  of  the  experiences  gained  by  the  F-15 
ISD  Team  personnel  and  their  "experimentation"  efforts  in 
establishing  F-15  tactics. 

For  air-to-ground  tactics,  F-15  will  not  be  a  primary  source 
of  experience  for  tactics  development  and  the  associated  training 
problems.  In  this  instance,  tactics  development  and  tactics 
training  for  the  A-10  and  F-4  will  prove  more  appropriate. 
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The  need  to  develop  tactics  impacts  upon  an  instructional 
system  in  terms  of  time-scheduled  events.  Tactics  instruction 
lies  at  the  heart  of  weapon  system  training.  It  therefore 
deserves  the  most  careful  attention  and  careful  preparation. 
Unfortunately  the  late  schedule  for  development  of  tactics  will 
make  it  difficult  to  expend  large  amounts  of  time  developing 
tactics  instruction.  This  requires  the  development  team  to 
design  flexibility  and  readiness  to  change  and  addition  to  the 
tactics  portions  of  the  F-16  pilot  course. 

3.2.7  CCTS  and  Continuation  Training 

The  results  of  the  review  of  previous  ISD  efforts  show  that 
establishing  the  line  of  division  between  what  will  be  taught  in 
formal  (CCT)  training  and  what  will  fall  to  the  responsibility  of 
continuation  training  can  be  difficult.  Every  advantage  should 
be  taken  through  additional  detailed  discussions  and  review  with 
the  F-15  and  A-10  ISD  Teams  to  learn  from  their  experience  in 
dealing  with  this  concern. 


3.2.8  Training  Alternatives  While  Awaiting  the  Simulator 

The  results  show  that  both  A-10  and  the  F-15  systems 
experienced  problems  training  certain  tasks  without  the 
availability  of  a  simulator  at  the  beginning  of  their  programs. 
This  same  situation  will  exist  for  F-16.  Additional,  more 
detailed  studies  should  be  made  of  the  methods  the  A-10  and  F-15 
ISD  Teams  employed,  such  as  the  video  tape  developed  for  the  F-15 
INS  system,  in  order  to  take  advantage  of  their  successful 
solutions  to  problems  that  can  be  expected  in  the  design  of  the 
F-16  training  program. 


3.2.9  STAN/EVAL  and  Objective  Student  Evaluation 

The  review  results  clearly  show  the  lack  of  objective 
student  evaluation,  especially  in  flying  tasks.  STAN/EVAL  was 
clearly  pointed  out  as  the  only  area  where  criteria  are  clearly 
specified.  Close  cooperative  efforts  should  be  established  early 
between  the  F-16  Team  and  STAN/EVAL  in  the  area  of  CRO  and  CRT 
development . 

As  CROs  and  CRTs  are  established  in  the  F-16  Training 
Program  procedures  need  to  be  developed  for  a  joint  F-16  ISD/Stan 
Eval  review  to  insure  a  common  agreed  upon  evaluation  baseline. 


Additionally,  in  the  design  of  the  F-16  Performance 
Measurement  System,  careful  attention  must  be  given  to  the  issues 
which  prevent  objective  evaluation  of  student  performance:  lack 
of  agreement  on  performance  criteria,  lack  of  sufficient  IP 
training,  lack  of  adherence  to  established  judging  procedures, 
and  demotivating  aspects  of  evaluation  for  IPs. 


